Viewing three dimensional digital slides

ABSTRACT

Systems and methods for retrieving, manipulating, and viewing 3D image objects from 3D virtual microscope slide images (“3D digital slides”) are provided. An image library module provides access to the imagery data in a 3D digital slide and constructs 3D image objects that are coextensive with the 3D digital slide or a 3D sub-portion thereof. From within the 3D image object, cross layer planar views spanning various depths of the 3D digital slide are constructed as well as 3D prisms and other shaped image areas. The image library module allows a 3D image object to be sliced into horizontal and vertical views, skewed cross layer views and regular and irregular shaped 3D image areas for viewing by a user.

RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 12/871,676 filed Aug. 30, 2010, issued May 29, 2012 as U.S.Pat. No. 8,189,891, which is a continuation of Ser. No. 11/341,097 filed27 Jan. 2006, issued Aug. 31, 2010 as U.S. Pat. No. 7,787,674, whichclaims priority to U.S. provisional patent application Ser. No.60/647,542 filed 27 Jan. 2005 and is related to U.S. patent applicationSer. No. 11/140,265, each of which is incorporated herein by referencein its entirety.

BACKGROUND

1. Field of the Invention

The present invention generally relates to virtual microscopy and moreparticularly relates to retrieval, manipulation, and viewing of threedimensional image objects from three dimensional virtual microscopeslide (“3D digital slide”) images.

2. Related Art

Conventional microscopes, using mechanical focusing mechanisms ofsufficient quality, are easily focused by an observer. The feedback loopbetween the eye/brain and the fingers is extremely rapid and accurate.If any part of the feedback loop is impaired, however, the process ofobtaining perfect focus becomes extremely difficult, time consuming andeven inaccurate. These impairments may arise from poor optical quality,insufficient or incorrect illumination, inaccuracies in the focusingmechanism, or even due to poor visual acuity of the operator.

More recently, conventional computer implemented microscope systems haveprovided the ability to remotely access high resolution images ofmicroscope slides for a number of years. Devices capable of creatingsuch images fall into two broad categories: (1) remote controlled (orrobotic) microscopes; and (2) imagers, which create some type ofelectronic or photographic image of some, or all, of the specimen. Eachtype has its advantages and disadvantages, many of which are unique tothe particular application.

Conventional imaging systems are capable of creating large enough imagesto capture an entire specimen at moderate or high magnification. Theseimages are called virtual microscope slides or virtual slides. Becauseof their extremely large size, virtual slides are usually limited to asingle focus level (“z-plane”). Many types of microscope specimens arevery thin so a single z-plane virtual slide is sufficient. Other typesof microscope specimens, however, are thicker than the depth of field ofthe objective lens.

Recent improvements in imaging systems include the ability to createvirtual slides with multiple z-planes. These virtual slide images arecalled three dimensional (“3D”) virtual slides and include imagery datafrom multiple focus levels throughout the thickness of the microscopespecimen. What is needed is a system and method for extracting threedimensional image objects from the image data in a three dimensionalvirtual slide for viewing by a user.

SUMMARY

Accordingly, the present invention provides systems and methods forretrieving, manipulating, and viewing 3D image objects from 3D digitalslide images. An image library module provides access to the imagerydata in a 3D digital slide and constructs 3D image objects that arecoextensive with the 3D digital slide or a 3D sub-portion thereof. Fromwithin the 3D image object, cross layer planar views spanning variousdepths of the 3D digital slide are constructed as well as 3D prisms andother shaped image areas. The image library module allows a 3D imageobject to be sliced into horizontal and vertical views, skewed crosslayer views and regular and irregular shaped 3D image areas for viewingby a user.

Other features and advantages of the present invention will become morereadily apparent to those of ordinary skill in the art after reviewingthe following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of the present invention, both as to its structure andoperation, may be gleaned in part by study of the accompanying drawings,in which like reference numerals refer to like parts, and in which:

FIG. 1 is a block diagram illustrating an example side view of amicroscope slide and specimen according to an embodiment of the presentinvention;

FIG. 2 is a block diagram illustrating an example side view of amicroscope slide and specimen with z-planes according to an embodimentof the present invention;

FIG. 3A is a block diagram illustrating an example z-plane according toan embodiment of the present invention;

FIG. 3B is a block diagram illustrating an example blocked z-planeaccording to an embodiment of the present invention;

FIG. 4A is a block diagram illustrating an example layer according to anembodiment of the present invention;

FIG. 4B is a block diagram illustrating an example blocked layeraccording to an embodiment of the present invention;

FIG. 5 is a block diagram illustrating an example image object accordingto an embodiment of the present invention;

FIG. 6 is a block diagram illustrating example image object layersaccording to an embodiment of the present invention;

FIG. 7A is a block diagram illustrating an example image object andplanar view according to an embodiment of the present invention;

FIG. 7B is a block diagram illustrating an example image object andcross layer view according to an embodiment of the present invention;

FIG. 7C is a block diagram illustrating an example image object andvertical cross layer view according to an embodiment of the presentinvention;

FIG. 7D is a block diagram illustrating an example image object andcross layer prism according to an embodiment of the present invention;

FIG. 8 is a block diagram illustrating an example networked systemaccording to an embodiment of the present invention;

FIG. 9 is a flow diagram illustrating an example method for constructinga planar view from a three dimensional image object according to anembodiment of the present invention;

FIG. 10 is a flow diagram illustrating an example method forconstructing a cross layer view from a three dimensional image objectaccording to an embodiment of the present invention;

FIG. 11 is a flow diagram illustrating an example method forconstructing an image object from a 3D virtual slide according to anembodiment of the present invention; and

FIG. 12 is a block diagram illustrating an exemplary computer system asmay be used in connection with various embodiments described herein.

DETAILED DESCRIPTION

Certain embodiments as disclosed herein provide for systems and methodsfor retrieving, manipulating, and viewing 3D image objects from 3Dvirtual microscope slide images. For example, one method as disclosedherein allows for a 3D image object to be obtained from a 3D virtualslide and the 3D image object can be rotated, flipped, and sliced toprovide a user with a complete 3D interrogation of a tissue sample.

After reading this description it will become apparent to one skilled inthe art how to implement the invention in various alternativeembodiments and alternative applications. However, although variousembodiments of the present invention will be described herein, it isunderstood that these embodiments are presented by way of example only,and not limitation. As such, this detailed description of variousalternative embodiments should not be construed to limit the scope orbreadth of the present invention as set forth in the appended claims.

Introduction

In one embodiment of virtual slide viewing, an image library module mayprovide access to a virtual slide image object. The image library modulemay be a programmed or programmable object implemented in software,hardware, or some combination of the two. For example, the image librarymodule may encapsulate access to a virtual slide. Accordingly, allaccess to a virtual slide may goes through this module, either locally(access to disk files) or remotely (access to an image server machine).An image server may even use the image library module to access diskfiles. Virtual slides can contain 3D data, in which multiple imageplanes are provided at a series of Z values. These multiple image planesmay occur over an entire scanned tissue area, or selected regions may becaptured in this way.

Image data that results from multi-planar acquisition is truly threedimensional. It can be perceived as a three dimensional solid which isdivided up into cubical pixels. Image planes that were not physicallycaptured can be synthesized between captured image planes usinginterpolation, in the same way that the resolution in X or Y can beincreased by bilinear or bicubic sampling.

In one embodiment, the image library module employs the central conceptof a “view”. A view is a two-dimensional rectangular region derived fromthe image object. The image object may be the entire 3D virtual slide orsome portion thereof. Software that wishes to display or process datafrom an image object retrieves data from the object as a view. Forexample, various view parameters are established and provided to theimage library module that in response retrieves and provides the view.

For two dimensional image objects all views are co-planar with the imageobject's plane. A view may begin at any X,Y location in the imageobject, have any desired dimensions (width and height), and may bescaled to any desired resolution (“zooming”). Views can also be rotatedrelative to the original image.

Even for three dimensional image objects, a view remains two-dimensionaland corresponds to a slice through a three dimensional space. Thesimplest possible case is that two-dimensional views are perpendicularto the Z axis of the image object. In this case views may begin at anyX,Y location in the image object, have any desired dimension (width andheight), and may be scaled to any desired resolution (“zooming”). Inaddition the view's plane may be located at any desired location alongthe Z axis (“focusing”).

More complicated cases arise when the view plane is not perpendicular tothe Z axis. For example, arbitrary orientation for views can beprovided, thus enabling full 3D viewing. Additionally, orthogonaltransformations (rotations and flips) enable “side views” and otherinteresting capabilities. The ability to tilt a view slightly from theZ-axis can provide a “parallax” effect. Note that regardless of the vieworientation, there is always an axis perpendicular to the view, and theview plane may be located at any desired point along this axis(“focusing” can be performed along axes other than Z). To support 3-Dimage objects, the image library module may provide a well definedapplication programming interface (“API”) that extends a number offeatures and functionality to applications that include the module.

Images have attributes that are determined at acquisition time and canbe queried via the API. Among these attributes are overall width (X) andheight (Y). Three dimensional image objects also have depth (Z). Forconsistency the units of depth are pixels, even though these don't havea physical dimension when an image is acquired. The resolution of animage object may also be retrieved in order to convert pixel dimensionsto physical units such as microns. For practical reasons image objectsmay have various resolutions pre-rendered, either by acquisition throughdifferent power objective lenses or through interpolation.

The image library module enables the attributes of each physical image(width, height, zoom relative to base) to be queried. Three dimensionalimage objects may have multiple z-planes pre-rendered (e.g.,corresponding to the actual planes which were captured). Thecharacteristics of these z-planes (width, height, zoom relative to base,and Z-location) can also be queried.

A view has attributes that can be modified through the image librarymodule. These attributes include X and Y offsets and width and height,expressed in pixels relative to the current view resolution, and zoom,expressed as a floating point scale value. Views of 3D image objectsalso have a focus attribute, expressed as a signed floating point value.In one embodiment, a zero value for focus can be the center of the imageobject, and the focus values can range from −1 to +1, with negativevalues designating planes closer in the view direction.

In addition to two-dimensional views through the 3D image object, it mayalso be useful to retrieve regular and irregular shapes such as prismsand polygons from a 3D image object. These shapes are three-dimensionalviews. In one embodiment, a prism can have all the attributes of a view,and additionally has a specified depth. As with image objects, prismscan be thought of as having cubic pixels, oriented to the view axiswhich defines the prisms' orientation.

A clarification regarding the term “depth” as applied to images will behelpful at this juncture. In conventional terminology, images have aproperty called bits-per-pixel. This is sometimes referred to as “colordepth.” Color depth is not the same as the “image depth” attribute of 3Dimages. Bits-per-pixel is a much more accurate term than “color depth”but both terms are used interchangeably by workers in the field ofdigital imaging. The color depth (bits-per-pixel) determines theprecision with which color values are stored. For example, with respectto color virtual slides, usually 24 bits per pixel are stored, namely 8bits for each of the red, green, and blue color channels. Each pixeltherefore requires 3 bytes.

D objects have depth (Z), as well as lateral extent (X,Y). The “imagedepth” is the number of pixels in the Z-direction and corresponds to theoverall change in the Z value. As an example, if we scan 10 z-planes at40×, each plane separated from each other by 1 micron, then the “imagedepth” is about 40 pixels, since there are about 4 pixels per micron.Note that there are 10 planes, but that the depth is 40 pixels. This isbecause the Z-sampling is coarser than the lateral (X,Y) pixel samplingof each image plane.

A 3D image object comprises one or more layers in the Z direction. Eachlayer may have depth (“thickness”), since it may be composed of multipleplanes; if a layer is composed of a single plane, then it has a depth of1 pixel. Each plane may in turn be composed of one or more co-planarimages.

Image objects may also be blocked. If they are, blocks are identified byX-offset and Y-offset within a layer. It is possible for some layers tobe blocked while others are not. If a layer is blocked, then all planeswithin that layer will have the same blocking offsets. However, eachlayer need not have the same blocking offsets as other layers. In oneembodiment, each of the blocks within a layer has the same depth.

Note that an image object is the entire 3D collection of images and maybe an entire virtual slide or a portion of a virtual slide.Additionally, a single image may be a portion of one layer of an imageobject. In alternative embodiments, a virtual slide may be a single filethat contains multiple images or a collection of individual files thateach comprise one image.

In one embodiment, the preferred file format used for storing 2D virtualslides allows for compression, in order to reduce file size. Thiscompression methodology can be extended to 3D image objects as well,resulting in a substantial saving of disk space and a correspondingimprovement in the performance of remote viewing. In 3D compression,there is also the ability to store a 3D image or section of a 3D imagein one disk file (e.g., using a multi-image disk file format such asTIFF). In such a case, output blocks are formed which have depth, thedisk file will have a single layer, and each block in the layer willhave depth. Advantageously, both 2D and 3D compression methods allow forblock-type compression, so that contiguous regions of image data can beretrieved efficiently by accessing only those blocks that are in commonwith the spatial region being retrieved.

Generally, when accessing and viewing image information, “fullresolution” denotes the resolution at which the data were acquired. At40×, using a typical acquisition device such as the Aperio ScanScope®,this corresponds approximately to 0.25 microns per pixel. When viewingit is often useful to zoom to lower resolutions. In order to facilitatezooming rapidly, sub-sampled versions of an image object can be createdand stored. These sub-sampled images may be stored as part of the imageobject. For a 3D image object each layer may be sub-sampled. Thus, avirtual slide image file may contain multiple images for each layer, atvarying resolutions.

It is advantageous to have the whole 3D image object in one file becauseit simplifies moving it around, renaming it, etc. It is alsoadvantageous to use a standard TIFF file format for a 3D image objectbecause it is standard and it is easy to implement through use of theexisting libtiff module, which works very well. One limitation, however,to using a single TIFF file is the 4 GB size limit. Since virtual slidestypically use lossy compression this size limit is usually not an issuefor 2D images, but 3D image object can be significantly larger.

As will be understood by those having skill in the art, there are avariety of ways to address the 4 GB size limit for TIFF files. Forexample, a single virtual slide file can be made bigger than 4 GB if afile format other than TIFF is used. Alternatively, image objects can bestored as multiple files. The multiple file approach may be preferablebecause it can be made backward compatible with techniques already inuse and files larger than 4 GB have other inherent problems, forexample, they exceed the size of a digital versatile disc (“DVD”).

Description of the Figures

FIG. 1 is a block diagram illustrating an example side view of amicroscope slide 100 and specimen according to an embodiment of thepresent invention. In the illustrated embodiment, the slide 110 supportsa sample of tissue 100 (also referred to herein as a specimen). Thetissue 100 may be any type of microscope slide specimen such as blood orother substances.

Although tissue samples such as tissue 100 on a microscope slide appearflat, at the micron and sub-micron resolution level they may actuallyhave a considerable amount of thickness. The thickness of samples canvary from 0.5 micron to 10 microns. To digitally capture medicallyimportant information on a thick specimen, an objective lens is focusedthrough the specimen at varying focal depths. At each focal depth, animage plane is captured, for example, an area of a specimen that is 10microns thick may have 10 separate image planes. Because the thicknessof the tissue 100 may not be uniform across the slide 110, sets of imageplanes may be acquired from various locations on the specimen. Eachindividual image plane corresponding to a thick region of the specimenis called a “z-plane.”

FIG. 2 is a block diagram illustrating an example side view of amicroscope slide 110 and tissue specimen 100 with z-planes 210 accordingto an embodiment of the present invention. In the illustratedembodiment, the z-planes 210 are identified at a location on thespecimen that is 10 microns thick. As will be understood, the tissue 100may have a relatively uniform thickness or may have peaks and valleys.In the illustrated embodiment, there are 10 z-planes 210. In thisparticular embodiment, the focus depth of the objective lens is 1 micronso that 10 contiguous Z-planes are captured with a 1 micron separationbetween each z-plane and collectively cover the entire 10 micronthickness of the specimen. In alternative embodiments, the depth of asingle Z-plane may be more or less than 1 micron and the variousz-planes maybe be spaced apart rather than being contiguous.

The lateral (X,Y) dimensions of each z-plane image can be determined bythe acquisition hardware, such as the size of a camera's CCD array. Inone embodiment the dimensions of the CCD array may be 2048×1536 pixels.A z-plane image may also be a combination of several fields of viewcombined together. The number and spacing of the z-plane images areconfigurable, and may be optimized according to the sample type andnature of the medical application. In one embodiment, a typical threedimensional virtual slide 5-10 z-plane images spaced 0.25-1.0 micronapart.

FIG. 3A is a block diagram illustrating an example z-plane 120 accordingto an embodiment of the present invention. In the illustratedembodiment, the z-plane 120 has a height 130 and a width 140. A z-plane120 may cover the entire planar area of a virtual slide or it may covera portion of a virtual slide image.

FIG. 3B is a block diagram illustrating an example blocked z-plane 150according to an embodiment of the present invention. In the illustratedembodiment, each block of the blocked z-plane 150 has a block height 160and a block width 170. In one embodiment, all blocks in blocked z-plane150 have the same block height 160 and the same block width 170.Advantageously, each block in a blocked z-plane 150 may be a separatelycaptured image or it may be an image area cut from the larger z-planeand stored as an individual block or compressed as an individual block.In one embodiment, the various blocks of a blocked z-plane 150 arestored in a single file.

FIG. 4A is a block diagram illustrating an example layer 180 accordingto an embodiment of the present invention. In the illustratedembodiment, the layer 180 may comprise a series of contiguous z-planes,providing the layer with a layer depth 190. The layer 180 also has aheight (not shown) and a width (not shown). Each of the contiguousz-planes may have been separately captured by an image acquisitiondevice, or alternatively some or all of the contiguous z-planes may havebeen interpolated from a z-plane image that was captured by an imageacquisition device.

FIG. 4B is a block diagram illustrating an example blocked layer 200according to an embodiment of the present invention. In the illustratedembodiment, each block of the blocked layer 200 has a block depth 210.Each block also has a block height (not shown) and a block width (notshown). In one embodiment, all blocks in blocked layer 200 have the sameblock height, the same block width, and the same block depth 210. Aswith the blocked plane, a block in the blocked layer 200 may be acombination of separately captured contiguous images. Alternatively, ablock may be a prism cut from the larger layer and stored as anindividual block or compressed as an individual block. In oneembodiment, the various blocks of a blocked layer 200 are stored in asingle file.

Notably, with respect to FIGS. 3A, 3B, 4A and 4B, the various planes andblocked planes and layers and blocked layers may be physical attributesof an image stored in single file or stored in multiple files.Alternatively the various planes and blocked planes and layers andblocked layers may be logical attributes of an image stored in singlefile or stored in multiple files. Alternative ways to physically orlogically arrange and store the image data of a virtual slide will beunderstood by those having skill in the art and are contemplated for usewith the present invention.

FIG. 5 is a block diagram illustrating an example image object 230according to an embodiment of the present invention. In the illustratedembodiment, the image object 230 has a width 240, a height 250, and adepth 260. The image object 230 may be an entire virtual slide or may bea portion of a virtual slide. The image object 230 may be filled withcontiguous image data or it may include various planes and layers ofimage data that are not contiguous and therefore define empty spacebetween them.

In one embodiment, an image object 230 is defined by its location in anX, Y, and Z coordinate system for a 3D virtual slide. The X, Y, and Zvalues can provide an initial offset for a first corner of the imageobject 230. The shape and size of the resulting image object 230 maythen be determined by its height 250, width 250 and depth 260 values.Accordingly, the image data for the complete image object 230 may thenbe obtained from the 3D virtual slide from actual scanned z-planes aloneor in combination with interpolations of scanned z-planes to fill in thespace between.

Alternative methods for defining the shape and size of an image object230 may also be employed. Advantageously, an image object 230 may bespherical or irregularly shaped and defined by appropriate equations, aswill be understood by those skilled in the art. For example, an imageobject 230 may be coextensive with a single cell in the tissue sample orsome other region of interest so as to provide a user with a 3D image ofjust the targeted region that can then be rotated and sliced forthorough interrogation.

FIG. 6 is a block diagram illustrating example image object layers 180according to an embodiment of the present invention. In the illustratedembodiment, the layers 180 of image object 230 include a z-plane 120, ablocked z-plane 150, a layer 180, and a blocked layer 200.Advantageously, an image object 230 may include complete contiguousimage data in a series of contiguous z-planes that collectively make upa single layer that is co-extensive with the top surface of the imageobject 230 and the bottom surface of the image object 230.

In an alternative embodiment, the image object 230 (as stored in memory)may only include a single z-plane. When such an image object 230 isretrieved for viewing (e.g., by the image library module), the missingimagery data can be interpolated from a single plane of image data inthe virtual slide or between two planes of image data in the virtualslide in order to fill in the missing image information in the imageobject 230. An advantage of capturing non-contiguous z-planes of imagedata for a 3D virtual slide is that the resulting size of the singlefile virtual slide image is significantly reduced.

FIG. 7A is a block diagram illustrating an example image object 230 anda planar view 300 according to an embodiment of the present invention.In the illustrated embodiment, the image object 230 may be logicallyoriented to a 3D coordinate system that includes a zero focus 270 planethat extends through the center of the image object 230. In oneembodiment, the focus of the image object 230 is in the range of −1 to+1, with negative focus 280 values designating planes closer to theobjective lens during image capture and positive focus 290 valuesdesignating planes further from the objective lens during image capture.For example, a focus value of −1 in an image object 230 would be towardthe top surface of the tissue sample and a focus value of +1 in an imageobject 230 would be toward the bottom surface of the tissue sample.

As shown, the image object 230 includes a planar view 300. The planarview 300 has a zero focus value and an (X, Y) offset value relative tothe image object 230. Once an image object has been retrieved, variousplanar views such as planar view 300 may be displayed to a user whoprovides offset and focus values (X, Y, and Z). The planar view 300 maybe co-extensive with the height and width of the image object 230 or itmay be n completely or partially interior area.

FIG. 7B is a block diagram illustrating an example image object 230 anda cross layer view 320 according to an embodiment of the presentinvention. In the illustrated embodiment, the cross layer view 320 isplanar, but is skewed against the horizontal z-planes (not shown) thatcomprise the image object 230. Advantageously, various orientations,rotations, flips, and the like can be obtained from the image object 230as cross layer views 320. Note that regardless of the view orientation,there remains an axis that is perpendicular to the view. The illustratedview is referred to as a cross layer view because it extends acrossmultiple z-planes.

In one embodiment, in order to construct a cross layer view (which isplanar), imagery data from contiguous horizontal z-planes in the imageobject 230 at the appropriate height, width, and focus values areobtained and combined to form the cross layer view 320.

FIG. 7C is a block diagram illustrating an example image object 230 anda vertical cross layer view 330 according to an embodiment of thepresent invention. In the Illustrated embodiment, the vertical crosslayer view 330 is similar to the cross layer view previously describedin FIG. 7B except that it is orthogonal to the z-planes (not shown) thatcomprise the image object 230. Advantageously, arbitrary views such asvertical cross layer view 330 can be provided, which enables full 3Dviewing, including orthogonal transformations (rotations and flips),side views, and other interesting skews and tilts, for example toprovide a parallax effect. Although planar views that that have a viewplane that is not perpendicular to the Z axis are more complicated torender, once the 3D image object has been obtained, any sort of planarview can be retrieved and displayed to a user.

FIG. 7D is a block diagram illustrating an example image object 230 andcross layer prism 340 according to an embodiment of the presentinvention. As with planar views (recall that a view is a two dimensionalregion derived from the image object 230), prisms may also be derivedfrom the image object 230. These prisms can be in line with the focusplane or they can be skewed. Cross layer prism 340 is an example of askewed prism. As with the planar views all varieties, shapes, and sizesof prisms can be derived form the image object 230. In a simpleembodiment, a prism may be a cube. In a more complex embodiment, a prismmay be an irregular shaped 3D region within the image object 230 and,for example, be coextensive with a particular region of interest such asa single cell or cluster of cells. Additionally, once a prism has beenidentified and retrieved, in one embodiment views can be derived fromthe prism to provide vertical and horizontal and skewed cross sectionsof the prism to enable complete and thorough interrogation of the areaof interest.

A prism such as cross layer prism 340 can be defined within the imageobject 230 by the height, width, and depth of two or more corners of theprism or by the height, width, and depth of a first corner and theheight, width, and depth of the prism. Alternatively, an equation thatdefines the surface of the prism (e.g., if the surface is irregular orcurved, etc.) can be employed to obtain the image data for the prismfrom the image object 230.

FIG. 8 is a block diagram illustrating an example networked system 350according to an embodiment of the present invention. In the illustratedembodiment, the system 350 comprises an image acquisition device (“IAD”)360, an image data server (“IDS”) 370, and an image viewing workstation(“IVW”) 380. Each of these devices is associated with a respective datastorage area 365, 375, and 385.

The IDS 370 and IVW 380 are each optionally configured with an imagelibrary module (“ILM”) 390 that performs many of the functions relatedto obtaining, manipulating, and presenting views and prisms as describedherein. The ILM 390 may be a programmed object, a hardware device orsome combination of hardware and software. The IDS 370 may use the ILM390 to access files in data storage area 375. The ILM 390 may also beimplemented in combination such that a portion of its functionalityresides and operates on the IVW 380 and a portion of its functionalityresides and operates on the IDS 370. In one embodiment, all access to 3Dvirtual slide image data goes through the ILM 390.

To support 3D image objects and 3D virtual slides, the ILM 390 providesan interface (e.g., an API that extends a variety of features andfunctionality) to applications. The ILM 390 may be implemented as asoftware instance that can be incorporated into an application or as aservice hosted on the IDS 520, for example. Preferably, the variousfeatures and functionality of the ILM 390 are encapsulated and providedin a well defined API. For example, images 3D virtual slide images haveattributes that are determined at acquisition time and these attributesmay be queried through the ILM 390.

The devices 360, 370, and 380 are communicatively coupled to each othervia a network 395. Alternatively, the devices may be directly connectedto each other or connected to each other via separate networks. In oneembodiment, two or more of the devices may be modularly integrated intothe same physical device, in which case the communication betweenfunctional modules may take place through inter-process communication,shared memory, common files or the like.

The IAD 360 comprises hardware and software which can acquiretwo-dimensional images from microscope slides. The IAD 360 may acquireimages from the slide at any location within the slide (X,Y) and at anyfocal depth (Z).

The IAD 360 typically provides optics to magnify specimens on amicroscope slide to resolutions required for medical diagnostics byPathologists. The lowest resolution is typically 1.0 microns per pixel,(often called “10×”) and the highest resolution may be 0.1 microns perpixel (“100×”). Typical resolutions are usually 0.5 microns per pixel(“20×”) or 0.25 microns per pixel (“40×”). The numerical aperture (“NA”)of the optics is typically 0.5 or 0.75, yielding a focal depth of 0.5micron to 1 micron, allowing contiguous z-planes in tissue samplesthicker than 1 and 2 microns, respectively.

Specimens on a microscope slide usually have an overall dimension ofbetween 10 and 30 mm per side. For example, a typical sample could covera region of 20×15 mm. Such a sample, when imaged fully at a resolutionof 0.25 microns per pixel (i.e., 40×), yields an image that has80,000×60,000 pixels. Accurately obtaining full slide images istechnically difficult, but may be accomplished with instruments such asthe Aperio ScanScope® that is described in detail in U.S. Pat. No.6,711,283 (Soenksen), which is incorporated herein by reference in itsentirety.

The IDS 370 comprises a server computer and software which stores andenables access to two- and three-dimensional images acquired by the IAD360. The IDS 370 can be located near the IAD 360 on a local-area network395. In an alternative embodiment it may be desirable for the IDS 370 tobe remote from the IAD 360 and connected to viewing stations and otheraccess devices via a wide-area network 395. The IDS 370 may include theILM 390 as appropriate for delivery of the 3D virtual slide image data.

The IVW 380 comprises a personal computer and software which can displaytwo and three dimensional images acquired by the IAD 360 and retrievedfrom the IDS 370. The IVW 380 may communicate across a local areanetwork or wide area network 395 with the IAD 360 and IDS 370, usingstandard computer network technology such as TCP/IP. Image data capturedat the IAD 360 may be stored in data storage area 365 or stored by theIDS 370 in data storage area 375. The image data may then be laterviewed using an IVW 380. The IVW 380 may include the ILM 390 asappropriate for obtaining, manipulating, and viewing of the 3D virtualslide image data. An ILM 390 may be deployed both on the IVW 380 and theIDS 370 to provide 3D image data to a user at the IVW 380.

The network 395 may be a local area network, a wide area network, awired or wireless network, or any combination of networks. On suchcombination of networks is commonly referred to as the Internet. Network395 may be the Internet or any sub-combination of the networks thatcomprise the Internet. Network 395 may also be a private network or apublic network or any combination of networks.

FIG. 9 is a flow diagram illustrating an example method for constructinga planar view from a three dimensional image object according to anembodiment of the present invention. In one embodiment, this method maybe carried out on an image data server by an image library module, suchas those previously described with respect to FIG. 8. Initially, in step400, the ILM receives a request for a view. The request preferablyprovides certain demographic information about the requested view, forexample the resolution of the image data and the offset and focus of theview along with the height and width of the view. Additional parametersabout the view such as information to help sufficiently identify a viewor prism may also be provided in the request, for example the type ofimage being requested (e.g., prism, cross layer view, standard view).

Once the identification information has been parsed out of the request,the ILM next determines whether the request is for a prism, as shown instep 405. If a prism has been requested, the ILM proceeds to step 410where it constructs the prism. FIG. 11 (described later) provides adetailed method for constructing a prism. If the request is not for aprism, the ILM next determines in step 415 if the request is for a crosslayer view. If a cross layer view has been requested, the ILM proceedsto step 420 where it constructs the cross layer view. FIG. 10 (describedlater) provides a detailed method for constructing a cross layer view.

If the request is not for a cross layer view, then in step 425 the ILMidentifies the offset for the view and in step 430 the ILM identifiesthe focus for the view in step 430. Next, in step 435 the ILM accessesthe imagery data in the identified z-plane and then in step 440constructs the planar view, for example, using the offset and focusvalues in combination with the height and width values to crop theimagery data of the entire identified z-plane to just the requestedview.

FIG. 10 is a flow diagram illustrating an example method forconstructing a cross layer view from a three dimensional image objectaccording to an embodiment of the present invention. In one embodiment,this method may be carried out on an image data server by an imagelibrary module, such as those previously described with respect to FIG.8. Initially, in steps 455, 460, and 465, the ILM identifies first,second, and third points on the requested cross layer plane. Thesepoints may be identified by one of or a combination of the parametersincluded in the request. For example, the points may be derived based onthe offset and the focus value for first, second, and third corners ofthe cross layer plane.

Next, in step 470, the ILM calculates the cross layer plane based on thethree identified points. Once the cross layer plane has been calculated,in step 475 the ILM accesses the imagery data in the cross layer plane.In one embodiment, in order to access this imagery data, the ILM mayhave to access imagery data in a plurality of z-planes that areperpendicular to the Z axis. Alternatively, the ILM may be able todirectly access the imagery data in the cross layer plane and thenconstruct the cross layer view, as shown in step 480. In constructingthe cross layer view, a portion of the imagery data for the completecross layer plane in the image object may be discarded.

The above described method may also be used to construct a prism from animage object. For example, the top plane and the bottom plane of a prismcan be identified according to the above method and then the imagerydata in between those planes defines the requested prism. Accordingly, across layer prism or a more simple prism can be constructed.

Additionally, as will be understood by those skilled in the art, thegeneral outline constructing a prism can also be employed to construct3D views of other shapes such as spheres or irregular shapes. In thismethod, the surface of the desired 3D shape is first calculated withinthe image object and then the imagery data interior to the calculatedsurface is retrieved as the 3D view. In this description, the term prismis used primarily to mean regular 3D shapes with parallel opposing sidesbut it is also used intermittently to mean irregular shapes or sphericalshapes or other 3D shapes from within an image object that can beretrieved and displayed as a 3D view. All such shapes are contemplatedby the breadth of the present invention.

FIG. 11 is a flow diagram illustrating an example method forconstructing an image object from a 3D virtual slide according to anembodiment of the present invention. In one embodiment, this method maybe carried out on an image data server by an image library module, suchas those previously described with respect to FIG. 8. Initially, in step500 the ILM identifies the top and bottom surfaces of the image object.Next, in step 505 the ILM determines if the imagery data for the topplane exists in the 3D virtual slide or if it lies in an empty gapbetween scanned imagery data. If the top does not exist, as determinedin step 505, then in step 510 the ILM obtains the nearest z-plane ofimage data and interpolates the imagery data for the top surface, asillustrated in step 515.

After obtaining the imagery data for the top surface (from scanned dataor by interpolation), in step 520 the ILM next determines whether thereis imagery data for the bottom plane in the 3D virtual slide or if thebottom plane lies in an empty gap between scanned imagery data. If thebottom does not exist, as determined in step 520, then in step 525 theILM obtains the nearest z-plane of image data and interpolates theimagery data for the bottom surface in step 530. After obtaining theimagery data for the bottom surface (again, from scanned data or byinterpolation), the ILM can construct the image object. Additionalparameters used by the ILM to construct the image object may include theimage object height, width, and depth.

The above describe method may also be employed to construct a prism froman image object that is not completely populated with contiguous imagedata. Accordingly, when constructing the prism, the image data for thetop and bottom surfaces are either present in the image object orinterpolated from image data that is present. Next, the image databetween the top and bottom surfaces is filled in with data from theimage object or interpolated data where necessary. In this fashion, aregular or cross layer prism may be constructed from an image object.

FIG. 12 is a block diagram illustrating an exemplary computer system 550that may be used in connection with the various embodiments describedherein. For example, the computer system 550 may be used in conjunctionwith the IAD, IDS, or IVW devices previously described with respect toFIG. 11. However, other computer systems and/or architectures may beused, as will be clear to those skilled in the art.

The computer system 550 preferably includes one or more processors, suchas processor 552. Additional processors may be provided, such as anauxiliary processor to manage input/output, an auxiliary processor toperform floating point mathematical operations, a special-purposemicroprocessor having an architecture suitable for fast execution ofsignal processing algorithms (e.g., digital signal processor), a slaveprocessor subordinate to the main processing system (e.g., back-endprocessor), an additional microprocessor or controller for dual ormultiple processor systems, or a coprocessor. Such auxiliary processorsmay be discrete processors or may be integrated with the processor 552.

The processor 552 is preferably connected to a communication bus 554.The communication bus 554 may include a data channel for facilitatinginformation transfer between storage and other peripheral components ofthe computer system 550. The communication bus 554 further may provide aset of signals used for communication with the processor 552, includinga data bus, address bus, and control bus (not shown). The communicationbus 554 may comprise any standard or non-standard bus architecture suchas, for example, bus architectures compliant with industry standardarchitecture (“ISA”), extended industry standard architecture (“EISA”),Micro Channel Architecture (“MCA”), peripheral component interconnect(“PCI”) local bus, or standards promulgated by the Institute ofElectrical and Electronics Engineers (“IEEE”) including IEEE 488general-purpose interface bus (“GPIB”), IEEE 696/S-100, and the like.

Computer system 550 preferably includes a main memory 556 and may alsoinclude a secondary memory 558. The main memory 556 provides storage ofinstructions and data for programs executing on the processor 552. Themain memory 556 is typically semiconductor-based memory such as dynamicrandom access memory (“DRAM”) and/or static random access memory(“SRAM”). Other semiconductor-based memory types include, for example,synchronous dynamic random access memory (“SDRAM”), Rambus dynamicrandom access memory (“RDRAM”), ferroelectric random access memory(“FRAM”), and the like, including read only memory (“ROM”).

The secondary memory 558 may optionally include a hard disk drive 560and/or a removable storage drive 562, for example a floppy disk drive, amagnetic tape drive, a compact disc (“CD”) drive, a DVD drive, etc. Theremovable storage drive 562 reads from and/or writes to a removablestorage medium 564 in a well-known manner. Removable storage medium 564may be, for example, a floppy disk, magnetic tape, CD, DVD, etc.

The removable storage medium 564 is preferably a computer readablemedium having stored thereon computer executable code (i.e., software)and/or data. The computer software or data stored on the removablestorage medium 564 is read into the computer system 550 as electricalcommunication signals 578.

In alternative embodiments, secondary memory 558 may include othersimilar means for allowing computer programs or other data orinstructions to be loaded into the computer system 550. Such means mayinclude, for example, an external storage medium 572 and an interface570. Examples of external storage medium 572 may include an externalhard disk drive or an external optical drive, or and externalmagneto-optical drive.

Other examples of secondary memory 558 may include semiconductor-basedmemory such as programmable read-only memory (“PROM”), erasableprogrammable read-only memory (“EPROM”), electrically erasable read-onlymemory (“EEPROM”), or flash memory (block oriented memory similar toEEPROM). Also included are any other removable storage units 572 andinterfaces 570, which allow software and data to be transferred from theremovable storage unit 572 to the computer system 550.

Computer system 550 may also include a communication interface 574. Thecommunication interface 574 allows software and data to be transferredbetween computer system 550 and external devices (e.g. printers),networks, or information sources. For example, computer software orexecutable code may be transferred to computer system 550 from a networkserver via communication interface 574. Examples of communicationinterface 574 include a modem, a network interface card (“NIC”), acommunications port, a PCMCIA slot and card, an infrared interface, andan IEEE 1394 fire-wire, just to name a few.

Communication interface 574 preferably implements industry promulgatedprotocol standards, such as Ethernet IEEE 802 standards, Fiber Channel,digital subscriber line (“DSL”), asynchronous digital subscriber line(“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrateddigital services network (“ISDN”), personal communications services(“PCS”), transmission control protocol/Internet protocol (“TCP/IP”),serial line Internet protocol/point to point protocol (“SLIP/PPP”), andso on, but may also implement customized or non-standard interfaceprotocols as well.

Software and data transferred via communication interface 574 aregenerally in the form of electrical communication signals 578. Thesesignals 578 are preferably provided to communication interface 574 via acommunication channel 576. Communication channel 576 carries signals 578and can be implemented using a variety of wired or wirelesscommunication means including wire or cable, fiber optics, conventionalphone line, cellular phone link, wireless data communication link, radiofrequency (RF) link, or infrared link, just to name a few. Computerexecutable code (i.e., computer programs or software) is stored in themain memory 556 and/or the secondary memory 558. Computer programs canalso be received via communication interface 574 and stored in the mainmemory 556 and/or the secondary memory 558. Such computer programs, whenexecuted, enable the computer system 550 to perform the variousfunctions of the present invention as previously described.

In this description, the term “computer readable medium” is used torefer to any media used to provide computer executable code (e.g.,software and computer programs) to the computer system 550. Examples ofthese media include main memory 556, secondary memory 558 (includinghard disk drive 560, removable storage medium 564, and external storagemedium 572), and any peripheral device communicatively coupled withcommunication interface 574 (including a network information server orother network device). These computer readable mediums are means forproviding executable code, programming instructions, and software to thecomputer system 550.

In an embodiment that is implemented using software, the software may bestored on a computer readable medium and loaded into computer system 550by way of removable storage drive 562, interface 570, or communicationinterface 574. In such an embodiment, the software is loaded into thecomputer system 550 in the form of electrical communication signals 578.The software, when executed by the processor 552, preferably causes theprocessor 552 to perform the inventive features and functions previouslydescribed herein.

Various embodiments may also be implemented primarily in hardware using,for example, components such as application specific integrated circuits(“ASICs”), or field programmable gate arrays (“FPGAs”). Implementationof a hardware state machine capable of performing the functionsdescribed herein will also be apparent to those skilled in the relevantart. Various embodiments may also be implemented using a combination ofboth hardware and software.

Furthermore, those of skill in the art will appreciate that the variousillustrative logical blocks, modules, circuits, and method stepsdescribed in connection with the above described figures and theembodiments disclosed herein can often be implemented as electronichardware, computer software, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled persons can implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the invention. In addition, the grouping of functions within amodule, block, circuit or step is for ease of description. Specificfunctions or steps can be moved from one module, block or circuit toanother without departing from the invention.

Moreover, the various illustrative logical blocks, modules, and methodsdescribed in connection with the embodiments disclosed herein can beimplemented or performed with a general purpose processor, a digitalsignal processor (“DSP”), an ASIC, FPGA or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,or any combination thereof designed to perform the functions describedherein. A general-purpose processor can be a microprocessor, but in thealternative, the processor can be any processor, controller,microcontroller, or state machine. A processor can also be implementedas a combination of computing devices, for example, a combination of aDSP and a microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

Additionally, the steps of a method or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumincluding a network storage medium. An exemplary storage medium can becoupled to the processor such the processor can read information from,and write information to, the storage medium. In the alternative, thestorage medium can be integral to the processor. The processor and thestorage medium can also reside in an ASIC.

The above description of the disclosed embodiments is provided to enableany person skilled in the art to make or use the invention. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles described herein can beapplied to other embodiments without departing from the spirit or scopeof the invention. Thus, it is to be understood that the description anddrawings presented herein represent a presently preferred embodiment ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It is furtherunderstood that the scope of the present invention fully encompassesother embodiments that may become obvious to those skilled in the artand that the scope of the present invention is accordingly limited bynothing other than the appended claims.

The invention claimed is:
 1. A computer implemented method forconstructing a three dimensional view of digital slide imagery data froma portion of an image object including scanned image data from aplurality of image planes, where one or more processors are programmedto perform the steps comprising: identifying a first point in an imageobject having a first height, a first width, and a first focus value;identifying a second point in the image object having a second height, asecond width, and a second focus value; identifying a third point in theimage object having a third height, a third width, and a third focusvalue; constructing a first plane in the image object from said first,second, and third points; and providing a cross layer three dimensionalview from said first plane, wherein the cross layer three dimensionalview comprises contiguous image information from at least two scannedimage planes with interpolated image information presented between atleast two of said scanned image planes.
 2. The method of claim 1 furthercomprising interpolating image information between scanned image planesof imagery data.
 3. The method of claim 1 further comprising providing across layer three dimensional view from said first plane, wherein thecross layer three dimensional view includes scanned image informationfrom at least three scanned image planes with interpolated imageinformation presented between said scanned image information.
 4. Themethod of claim 1 further comprising providing a cross layer threedimensional prism of contiguous virtual slide information, wherein thecross layer three dimensional prism includes scanned image informationfrom at least two scanned image planes with interpolated imageinformation between said scanned image information.